Focusable virtual microscopy apparatus and method

ABSTRACT

A virtual microscope slide includes images of a specimen for a given level of optical magnification which are associated and stored in a data structure. The forming of the data structure having the multiple Z-plane images preferably includes automatically focusing at a principal reference focal plane and capturing and digitizing an optically magnified reference Z-image and then shifting the specimen relative to the lens system by a predetermined increment to capture and digitize another Z-plane image. Preferably, a multiple sequence of Z-plane images above and below the reference image and captured and digitized. For ease of retrieval and use, each reference image has its associated Z-plane images are formed in a stack that is sent over the Internet, or Intranet to a local computer storage for quick retrieval when a viewer wants to mimic a focusing up or down to better view a detail in an image. Thus, the resultant images are retrieved and displayed such that a virtual focusing capability is available to the user. The images can be formed with overlapping fields of depth, adjacent fields of depth, or wholly separated fields of depth.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/549,797, filed Aug. 28, 2009, now U.S. Pat. No. 7,925,067, issuedApr. 12, 2011, which is a continuation of U.S. patent application Ser.No. 10/373,156, filed Feb. 24, 2003, now U.S. Pat. No. 7,596,249, issuedSep. 29, 2009, which claims the benefit of U.S. Provisional ApplicationNo. 60/358,747, filed Feb. 22, 2002, which are incorporated by referenceas if fully set forth.

FIELD OF INVENTION

This invention relates to a method of, and an apparatus for, acquiringand constructing virtual microscope slides that include a Z-axis imagedimension across the entire virtual slide, from a specimen on a support,such as a glass microscope slide, such Z-axis image content beingrelative to multiple individual principal, or reference, image focalpositions across the glass microscope slide; and for storing, andtransferring the virtual microscope slide images including a coordinatedand seamless in the X, Y-plane of a Z-axis dimension, for viewing byanother to allow virtual focusing at a local or remote location.

BACKGROUND

Magnification of small objects using a microscope is well known.Microscopes facilitate magnification of small objects to thereby allowdetails of the small objects to be rendered visible. At any givenmagnification, a microscope has a corresponding field of view. Ingeneral, the greater the amount of magnification the smaller thecorresponding field of view relative to the object. Similarly, and asrepresented in FIG. 1, at any given focal distance, a microscopeobjective lens (10) has a corresponding focal plane with a depth offield (11) (that is, a Z-axis range within which objects will appear tobe in focus). In general also, the greater amount of magnification thesmaller the corresponding depth of field relative to the object. Thecapture of single digital images of these microscope fields of view iswell known, and the art is experienced with the capture and display ofstacks of images at a single object position to record depth of fieldimage content. Such images are used for example in confocal microscopyinstruments to image through objects by varying the Z-axis focalposition of each image in the image stack at a single X, Y planarlocation.

In the early microscope technology, around 1750, microscope specimenswere placed between 2 small, thin circular glass plates, and mounted onlong ivory “sliders” that could be pulled back and forth in a slot underthe microscope objective lens. With today's technology the sliders havebeen replaced by rectangular glass “slides” as a mounting structure, theobject specimen is placed on the slide and sometimes covered by athinner glass “coverslip”. These glass slide mounting structures are notflat over their entire surface area, i.e., within the tolerances of thedepth of field of a common 40× to 100× microscope objective lens. Theyare thicker in some portions than in others and sometimes have a warp orcurvature. This creates a significant problem in the construction of avirtual microscope slide in contrast to taking a single field of viewimage. This is because in most instances the Z-axis focal plane of theobjective will not be positioned in the same cross sectioned portion ofthe specimen, and thus not be “in focus” across the entire surface ofthe slide, i.e., in adjacent planar X, Y field of views, withoutadjusting the specimen in the Z-axis dimension in some manner. Forexample, in the simple case of one end of the slide being thicker thanthe other end, all other factors being equal, and assuming the stagesupport is flat, this produces a slope across the slide with regard topositioning the same portion of the cross section of the specimen in theobjectives focal plane. This is not a problem for single field of viewmultiple Z dimension images because the slope is not apparent in thesmall field of view. Another aspect of this problem relates to the stagesupport. Stages commercially available are often not parallel and flatacross the complete working distance of the commonly used glassmicroscope slides. Also microtome sectioning does not produce uniformlythick sections. So in cross section the thickness of the specimen objectvaries. Thus the proper focal plane can vary from place to place on theslide from a multitude of factors. The focal distance position isdetermined by the microscope objective lens, and although the lens couldmove to adjust the focal plane position, it is common to move the stageplatform that holds the glass slide structure up and down in the Zdirection to obtain the optimal focal plane for a given specimenlocation and single field of view, or image tile. Thus, as is well knownin the art, the focal plane position in the Z-axis, relative to themicroscope slide planar surface and deposited specimen thereon variessubstantially from point to point for accurate focus in a give specimen.

Virtual microscope slides are also known. U.S. Pat. No. 6,272,235 B1(entitled Method and Apparatus for Creating a Virtual Microscope Slide),the contents of which are incorporated herein by this reference, teachesthe creation, storage and Internet or intranet display of virtualmicroscope slides. As taught therein, a virtual microscope slidetypically comprises a digitized magnified view of part or all of amicroscope slide and an object (such as a biological specimen) disposedthereon. Virtual microscope slides; when created, overcome limitationsof the microscope optical field of view restrictions; they have a datastructure for storing the digital images from different parts of theslide to enable the reconstruction of an X, Y planar view from compositeimage parts; and when viewed, overcome the limitations of the finitesize of computer terminal display screens, with Internet or intranetviewer software that seamlessly and rapidly allows the user to navigatefrom place to place in the virtual image, and to zoom the virtual imageto mimic changing of magnification with different microscope objectives.Prior art virtual slides allow computer viewing to mimic the viewing andinspection process obtained by looking through a real microscope withrespect to viewing abutted, aligned X, Y planar image views.

As taught in the aforesaid patent, the area of the object digitized iscomprised of multiple, adjacent, microscope objective optical fields ofview captured at a single Z-plane focal distance. In some casesthousands of microscope objective optical fields of view are recorded torepresent the virtual microscope slide. As taught in the aforesaidpatent, the individual digitized fields of view are referred to astiles. The chosen Z-plane object position varies for a given tile withthe X, Y location on the microscope slide, and as taught in theaforesaid patent, is obtained as a representative, reasonably optimum,focal position choice by an automatic focusing determination onindividual image fields, or by extension from previously determinedfocal positions of nearby image fields. The object is digitized and theresulting images stored in a data structure that allows for subsequentretrieval for review or image processing.

Because of the limitations of the microscope objective lens optics fieldof view, the capture event of virtual microscope slide tiles is alwaysrestricted to only a small part of the object in at least one planardimension. As further taught in the aforesaid patent, the digitalcapture was with a 3 color chip CCD sensor, which enabled the sameobject area sampled pixel point in and individual tile to be captured as3 identical color pixels, in register with each other. In an alternativeembodiment of a scanning method not taught in the aforesaid patent, aline sensor, e.g., with dimensions of 3×2098 pixels, could be used andmoved in one direction at a constant speed, and the sampling could beperformed to acquire a series of tiles of dimension 3×2098 stored incomputer memory to form a larger image segment. However, this imagesegment is still limited in one direction by the optical field of view,and subsequent adjacent tiled image segments are acquired to constructthe virtual microscope slide. In this case the 3 pixels at each givenposition along the line sensor provide different color sensing, thusthere is a small loss of color and spatial information with this method.As is known in the art, other combinations of sensor sampling can beobtained. However to construct a truly virtual microscope slide imagecapture that can be reconstructed to abut captured image portions, themethod must overcome the limitation of the very small optical field ofview in at least one dimension of the object plane of the microscopeobjective lens at high magnifications. Typically this is accomplished byeither moving the stage or the microscope objective to cover the objectarea and construct the digitized image data structure.

It may be further appreciated that the digitized image data structuremay be stored in numerous ways to facilitate future viewing. One methodmay be to simply store each capture event in a very large contiguousdigital memory or storage. In this case the subsequent viewing may beaccomplished by simply indexing this memory and displaying standard 2dimensional images, e.g., of X by Y pixel size, on a computer screen.However, with this method the virtual slide Internet server memoryrequirements become very large. As described in the aforesaid patent atiled data structure is more efficient of server memory and Internettransmission speed.

It is additionally taught in the aforesaid patent, that the standardcomputer video display will also only display a small portion of thetotal virtual slide at the original capture resolution, or highestmagnification. To overcome this, various methods of image data structureand storage have been described, and typically the viewer program canzoom in and out to display high and low magnification fields, and cancache portions of the virtual slide image data that have been previouslytransmitted from digital storage or an Internet server and viewed. Theviewer display programs must handle the indexing and addressing to bringin only the user requested image portions. Also, the virtual microscopeslide can be scrolled in various directions and thereby mimic movementof the object/slide with respect to the microscope objective lens. Suchvirtual microscope slides can be used for a variety of purposes,including education, training, and quantitative and qualitativeanalysis.

For many applications, such virtual microscope slides work well, andespecially with specimens that are of relatively uniform thickness andwith features of interest that tend to be within a single depth offield. Such virtual microscope slides solve the first of two significanttechnological issues of virtual microscopy; the first being the issue ofaligning small adjacent image segment views and displaying themseamlessly in X, Y registration. For any given level of magnification,the microscope can be automatically focused on such a specimen and thecorresponding single focal plane image digitally captured and stored forlater retrieval and use.

When the specimen exhibits significantly varying depth, however, and/orwhere features of interest are more widely spaced with respect to depth,prior art virtual microscope slides may contain images that are notfully focused with respect to one or more desired elements. This is thesecond major technical issue with virtual microscopy; the issue beingfinding the proper focal plane to represent the image in the firstplace, or alternatively including the Z-axis dimension across the entireslide and in so doing in either case, overcoming the problem of anon-flat microscope glass slide support and the problem of tissuesectioning and deposition irregularities that change the position of theoptimum focal plane relative to the planar X, Y surface of the glassslide. Consistent with the inherent problems of this second issue,obtaining stacks of Z-plane images in an uncoordinated fashion from manydifferent non-abutting object positions, without an integrated virtualslide data structure is both difficult to adequately store and retrieve,and to view in a coherent fashion in an Internet or internetenvironment. For example, and with reference to FIG. 2, a microscopeslide (21) can bear a specimen having portions (22) of relatively evendepth, or Z-axis position, and/or portions (23) that vary significantlywith respect to depth. While some portions (22) may reside within thedepth of field (11), other portions (26 and 27) that extend above orbelow the depth of field (11) will likely appear unfocused in theresultant image. Similarly, features of interest (24) that occur withinthe depth of field (11) will appear focused but features of interest(25) that are outside the depth of field range may appear unfocused.Regardless of whether such a virtual microscope slide is being usedacademically, for tissue microarray imaging, as in U.S. Pat. No.6,466,690 B2, or with diagnostic intent, unfocused elements often rendersuch an image unsuitable for the desired activity.

SUMMARY

In accordance with the present invention, there is provided a new andimproved method and apparatus for constructing, storing, and thenviewing virtual microscope slides from a microscope specimen thatincludes the capture of multiple Z-plane images to preserve depth offield image content. The improved method and apparatus also includesstoring the data structure of the individual tiled, or captured imagesin a format that includes the Z-plane images but is relative to a chosenoptimal image tile, allowing for full reconstruction of adjacent areasin multiple Z-planes, and enabling an Internet virtual microscope serverto efficiently transfer the virtual slide images with multiple Z-planesfor viewing by another at a remote location. This is achieved in thepreferred embodiment as a multiple Z-axis sequence of image captures,referenced by an automatically obtained chosen Z-axis focus position ofa single tile at a given X, Y position, as such scanning is taught inthe aforementioned patent. Multiple Z-plane images are captured aboveand below the given reference tile, and associated with it in the datastructure.

The preferred data structure is also provided with a proprietary virtualslide Internet/intranet Browser and generic component panel viewingprograms, e.g., an ActiveX component and Java Applet, all of which allowthe remote user to manipulate the Z-axis image dimension when viewingvirtual slide images, either in the proprietary Internet/intranetBrowser, or in the users own application programs or general purposeInternet/intranet Browsers. The data structure may be transmitted overthe Internet or intranet so that users may focus up or down at a givenobject position to view the virtual slide specimen throughout a Z-axisdepth, and thus bring objects and detail into focus that cannot be seenwith just one recorded Z depth of focus tile. In the preferredembodiment of this invention such viewing can be accomplished by movinga computer mouse wheel back and forth, or by moving through differentZ-axis images with computer keyboard up or down arrows. Further theviewing programs allow the user to scroll and to view neighboring imageareas of neighboring tiles and view the associated Z-axis images.

Turning now in greater detail to aspects of the invention, problems withachieving the additional Z-axis image content relative to the principalimage focal plane are overcome by the system of the invention. Thesystem includes a microscope stage which holds and supports the glassslide (21) at a certain fixed distance below the microscope objective(10), so that the specimen on the glass slide has an appropriate objectwithin the depth of focus (11) for the given microscope objective. Themicroscope stage is computer controlled by precision stepping motors inthe X, Y plane and also in the Z-axis dimension. Scanning in the X, Yplane with the preferred method of this invention occurs by moving thestage with the X, Y stepping motors precisely from one image field ofview to another to acquire image tiles. The step sizes for each x or ymovement occur in predetermined incremental step sizes so that the tilesabut and align with one another. Since the glass slide is held andsupported firmly by the stage, and the specimen is held firmly on theglass slide, the effect is to move the glass slide and thus new specimenparts into the field of view of the microscope objective. However, thecontent of the image is subject to the given depth of focus (11) of theobjective. Specimen parts in the field of view, but outside of the depthof focus region are not included in the image content. The microscopestage which supports the microscope slide is also controlled in theZ-axis direction so that it can move the specimen parts in a field ofview on the slide that are not in the Z-axis depth of focus region, intothe Z-axis depth of focus region as desired. Movement of the microscopestage in the Z-axis is computer controlled in digital increments ofZ-axis step size. Each digital unit represents the smallest incrementalstep possible. For example, in one automated microscope system, theOlympus BX61 (sold by Olympus America Inc. 2 Corporate Center Drive,Melville, N.Y. 11747, USA) with the internal motorized Z-drive, oneincrement represents 0.01 um. During the setup phase, prior to scaninitiation certain Z-axis step size parameters are defined for automaticfocus, and for a subsequent Z Stack image tile save procedure. For anygiven tile the Z Stack save procedure saves a set of 4 image tiles abovea given reference Z-axis position and 4 image tiles below that Z-axisreference position. Each image tile in the set is separated from thenext in the Z-axis dimension by the Z-axis step size parameter. Therelative reference position for each new field of view tile is obtainedby an iterative automatic focus procedure as follows. Upon moving to thenext tile, the Z-axis focus position is incrementally changed to go up 4times in automatic focus step sizes and acquire an image at each stepand then to go down in automatic focus step sizes and acquire an imageat each step. A focus contrast parameter is computed on each image. Theautomatic focus position is then determined by choosing the Z-axisposition associated with the largest value of the focus parameter fromthe reference image and the set of 8 image tiles. If the largest valueis at one end of the sequence, the procedure is recursively repeateduntil the largest value is found in the middle range of the sequence oftiles. This becomes the reference tile image. At that point the systemproceeds to use the Z-axis step size and execute the Z Stack saveprocedure. These Z stack image planes are added to the tiled image datastructure, and associated with the reference tile so that they can beeasily accessed for later retrieval and display. The same series ofevents is repeated for all field of views associated with the capture ofthe virtual microscope slide.

The step sizes chosen as input parameters for the scan relate to theZ-axis incremental resolution of the microscope system, to the chosenmicroscope objective lens, and to the requirements of the specimen,determined primarily by the sectioning thickness of tissue sections orthe smear thickness of blood or cell smears. For example, an incrementalZ-axis size of 0.01 um, with an automatic focus step size of 40 unitswould provide a travel range of 1.6 um up and 1.6 um down, for a totaltravel in one sequence of 3.6 um in a tissue section. This can becompared for example to a commonly used tissue section thickness of 5um. A Z stack step size of 20 would then similarly result in a focusrange of 1.8 um that could be examined virtually in 9 discreet anddifferent depth of field focal planes, according to the apparatusand—method of the invention.

It should be appreciated that the 2 step procedure of first determininga next relative focus position, and then recording the full chosen Zstack range allows for the compensation of irregularities caused bynon-flatness of the glass slide substrate, and by uneven tissuesectioning and deposition of clumped cells in blood and in smearpreparations. This preferred method including the recursive aspect, anddifferent adjustable Z-axis step sizes for the automatic focus, and thenfor the Z Stack capture, also enable a robust tracking up and downreference focal depth of field slopes in the specimen. The preferredmethod also allows for an efficient storage of image information thateffectively increases the usable image content in the Z-axis dimension.This is especially true for very thick specimens, such as plant materialmounted on a glass slide, or thick sections including whole mounts ofsmall organisms and insects. When used with the virtual slide Internetserver and viewer software the preferred method allows for efficientuser visual inspection and viewing of the additional Z-axis imagecontent

BRIEF DESCRIPTION OF THE DRAWING(S)

FIG. 1 comprises a depiction of a prior art microscope objective and acorresponding depth of field in the focal plane of the objective;

FIG. 2 comprises a prior art depiction of a specimen on a microscopeslide showing object detail in the specimen inside of and outside of thedepth of field;

FIG. 3 comprises a block diagram depiction of an embodiment configuredin accordance with the invention;

FIG. 3A comprises a window that allows the automatic focus setup stepsize parameter to be input.

FIG. 3B comprises a window that allows the Z Stack setup step sizeparameter to be input.

FIG. 3C comprises an example of a virtual slide folder with part of thedata structure showing the correspondence between the reference datastructure image tile, the Z-axis dimension focus data structure imagetiles, and the .ini data file for the given virtual slide.

FIG. 4 comprises a side elevation view of a specimen on a microscopeslide in an embodiment configured in accordance with the invention;

FIG. 5 comprises a depiction of overlapping depth of fields in anembodiment configured in accordance with the invention;

FIG. 6 comprises a depiction of non-overlapping depth of fields in anembodiment configured in accordance with the invention;

FIG. 7 comprises a side elevational view of another embodimentconfigured in accordance with the invention;

FIG. 8 comprises a top plan view of a composite virtual microscopeslide;

FIG. 9 comprises a perspective view of a symbolic model of a virtualmicroscope slide as configured in accordance with the invention;

FIG. 10 comprises a flow diagram configured in accordance with anembodiment of the invention; and

FIG. 11 comprises a flow diagram detail as configured in accordance withanother embodiment of the invention.

Skilled artisans will appreciate that elements in the figures areillustrated for simplicity and clarity and have not necessarily beendrawn to scale. For example, the dimensions of some of the elements inthe figures may be exaggerated relative to other elements to help toimprove understanding of various embodiments of the present invention.Some features may also be depicted in limited numbers and commonelements may be omitted for purposes of brevity and clarity.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 3 is a block diagram of a system according to the invention foracquiring a virtual microscope slide, that includes a Z-axis imagedimension across the entire virtual slide. The system includes amicroscope subsystem 15 with a digitally controlled stage platform 28for supporting the glass microscope slide 21. The digital stage platform28 can operate over a large number of increments to position the stagein the horizontal x and y plane with high precision. A glass microscopeslide or other substrate 21 is placed on the stage 28. The system alsoincludes a controlling computer system 32 with a keyboard 37, a mouse 38with a mouse wheel control 39, and a display monitor 40. The controllingcomputer system keyboard and mouse are used via the automatic focus stepsize setup window 12 to input the automatic focus step size parametersand the Z Stack step size setup window 13 is used to input the Z Stackstep size parameters.

FIG. 3A shows the focus setup step size 55 input control. Also in FIG.3A are shown the associated setup controls for the frequency of focus56, a threshold for controlling whether automatic focus should beperformed on a specific field of view, and a control 58 to manuallyincrement the Z-axis dimension to move the microscope stage 28 up ordown vertically in incremental units. As may be appreciated by theforegoing description, and the following descriptions, and as is wellknown in the art, the control of focus at high magnification and smalldepth of field is complicated and involves many variables. It is alsotime consuming if performed on every specimen image field of view inconstructing or capturing a virtual microscope slide image data set.Therefore, in the preferred embodiment and in the subsequently describedalternative embodiments there are additional control and setupparameters to overcome major variables and to enable a faster overallvirtual slide scan capture time. Some of these are seen in FIG. 3A. Forexample, the frequency of focus 56 parameter allows automatic focus tooccur on every adjacent field of view if it is set to 1, or on everyother field if it is set to 2, etc. In the following detaileddescription the assumption is that the frequency of focus is set to 1.However, if it is set to a higher number the reference tiles of field ofviews not focused take the default focus contrast value of the lastprevious focused image tile. In the alternate embodiments of theinvention, the reference tile position is sometimes obtained in adifferent fashion as described for those embodiments. There is asignificant speed of scan advantage related to not focusing on everyfield of view. However, on many specimens the disadvantage is that theresulting scan may not have an optimum depth of field position for thereference image. It may also be appreciated that one of the Z Stackimages may then offer a more optimum image for the final remote viewerof the images. Also, often there is not enough image structure in afield of view to obtain an automatic focus, for example in the case of asubstantially blank or empty field of view. In that case the control 57allows for a focus contrast threshold value input parameter that can bechecked to allow skipping such fields. Also, in that instance thedefault reference Z-axis position for the next image requiring focus isthe last previous focused image tile.

FIG. 3B shows the Z Stack step size 60 input control. It may beappreciated that there are a multitude of factors that would requirethis parameter to be changed for a specific specimen. However, the mostimportant of these is usually the magnification of the specificmicroscope objective lens being used, since each lens has a differentdepth of field specification, in combination with the type of specimenand estimated thickness of the specimen preparation. Also, shown in FIG.3B is a checkbox control 59 to either enable or disable the Z Stackimage save for a specific virtual microscope data capture scan.

According to the teachings of the aforesaid patent the computercontrolled microscope is moved to start a scan of the entire specimenobject 31 using the stage controller 14 to move the precision stage 28to a new objective lens 10 field of view to acquire an initial image atthat position and compute a focus contrast parameter on that image.According to the present invention the relative Z-axis referenceposition for the first new field of view image tile is obtained by aniterative automatic focus procedure. The controlling computer system 32sends the microscope subsystem 15 Z-axis control signals to change theZ-axis position control to move the stage incrementally to go up 4 timesin the automatic focus step size and then to go down 4 times inautomatic focus step size. At each incremental change in the Z-axisposition the image acquisition electronics 17 are controlled to acquirean image. A focus contrast parameter is computed on each image. Theautomatic focus Z-axis position is then determined by choosing theZ-axis position associated with the largest value of the focus parameterfrom the initial reference image and the set of 8 images. If the largestvalue is at one end of the sequence, i.e. the 4.sup.th image down or the4t image up from the reference image, that image becomes the referenceimage, and the procedure is recursively repeated until the largest focuscontrast value is found in the middle range of the sequence of images,i.e. not at either end image. This becomes the relative Z-axis referenceposition for the new field of view image. As explained more fully in thefollowing, the image tile associated with this relative Z-axis positionis then stored in the virtual slide data structure.

In the preferred embodiment of the invention the controlling computersystem is operated under a Windows Operating System (MicrosoftCorporation, Redmond, Wash., USA). Referring to FIG. 3A, the virtualmicroscope slide data structure is stored as a Windows Operating Systemfile folder where each tiled image is a .jpg image file with anincremental image name automatically assigned by the controllingcomputer systems software program. The .jpg image names are numbered sothat the first acquired image is called DA0.jpg, the second DA1.jpg, thethird DA3, etc. In FIG. 3C there is depicted a virtual slide data folder42 with portions of the data structure 43, 44, 45, 46, and 47 alsodepicted. The set of 9 image tiles 43, 45 and 46 named DA98 areassociated with a specific X, Y specimen image plane position, and anadjoining set 44 and 48, named DA99 are associated with another abuttedspecific X, Y specimen image plane position. The two reference tiles aredepicted as 43 and 44 for the data structure at the two X, Y locations.These tiles are in the automatic focus determined Z-axis position, andthe recorded .jpg image contains the depth of field image structureassociated with that Z-axis position and the field of view at therespective X, Y location.

During the system program operation to produce a virtual microscopeslide, the controlling computer system also creates an additional textinformation file of the Windows Operating System format .ini. Asdepicted in FIG. 3C, this file 47 is named FinalScan.ini. Among otherthings this file contains a list of names corresponding to eachreference tile in the virtual microscope data structure. For eachreference tile in the list, that tiles X, Y, and Z digital location istabulated. As taught in the aforementioned patent this information isthen used by the virtual Internet server and virtual microscope visualdisplay programs to abut and reconstruct the various tiled images toallow a remote viewer to view contiguous regions of virtual slideimages. It may be appreciated that the components of the data structureshown by example in FIG. 3C may be stored in a database or any otherform allowing rapid sequential access to the reference image and thefull Z Stack components. A novel aspect of this data structure is theclose association of these image components. This greatly facilitatesclient server interactions in remote Internet viewing. Since the thissubsequent viewing is through the limited X, Y planar view of an imagedisplay device, only a few reference tiles (and in certain limitedsituations only one reference tile) may be in the available user viewfor a focus request to the server. As will be appreciated in thefollowing description, this type of data structure facilitates rapidtransmission of Z-axis image content to the client computer. SomeInternet server computers facilitate serving very large images requiringzooming, by using a pyramid data structure where different levels ofimage zoom are pre-constructed from the original image and kept inmemory or virtual memory at one time. This requires a very large amountof memory when considering the requirement of keeping multiple planes ofsuch image structures, such as shown conceptually in FIG. 9. The datastructure of this invention is much more efficient when usedspecifically for virtual microscope slides with special viewingprograms, since it in essence is pre-constructed to serve reference andz Stack images rapidly from memory or digital disk storage in thesesmall reference and Z Stack image units.

After capturing a relative tile for the Z-axis position at a given X, Yspecimen plane position the system of the invention proceeds to use theZ-axis step size and execute the Z Stack save procedure. To accomplishthis, the controlling computer system 32 is directed to control theZ-axis positioning control 16 of the microscope subsystem 15 first tomove down the Z-axis in incremental Z-axis step sizes, and at each stepto acquire an image tile. These image tiles 45 are stored in the datastructure depicted in FIG. 3 by example for the data-structure set DA98.Secondly, the controlling computer system 32 is directed to control theZ-axis positioning control 16 of the microscope subsystem 15 to move upthe Z-axis in incremental Z-axis step sizes from the reference Z-axisposition, and at each step to acquire an image tile. These image tiles46 are stored in the data structure depicted in FIG. 3 by example forthe data structure set Da98.

The same series of events described above for the data structure captureof the tile set Da98 is repeated for all field of views associated withthe capture of the virtual microscope slide. For example, in FIG. 3 asthe data structure set Da99, and so forth.

The result of the above described preferred embodiment of the system ofthe invention is in effect to first factor out, or neutralize, theZ-axis irregularities in optimum focus position over the X, Y surface ofthe slide for the initial relative captured image tile, and then,secondly to create a set of cohesive Z-axis dimensioned captured imageplanes, where each plane relates to a different, real, physical depth offield position in the specimen. The first relative Z-axis positioninghas brought into parallel positioning capture, the optimum depth offield portions of each specimen, and the Z Stack capture has resulted inimage planes above and below that. This image sequence sampling can bereconstructed from the data structure storage elements 43, 44, 45, 46,and 48 when used with the X, Y location information stored in datastructure element 47. This reconstruction is depicted in an idealizedfashion as shown in FIG. 4 in cross section and in FIG. 9 in perspectiveschematic, as an aid in visualizing the resulting complete virtualmicroscope slide data structure. As described below, in fact only asmall portion of this is seen by the remote viewer at one time, becauseof the limitations of the image display of commonly used computerscreens. However, it is all available for viewing by scrolling andrequesting additional image tiles from a virtual microscope slideserver.

It will be appreciated by those familiar with the art that the abovepreferred description of the embodiment of the invention may be modifiedin other ways to enable the creation of a virtual microscope slide withZ-axis image dimension information. In this regard, an alternativemethod of practicing the invention is described. This method is moreapplicable for specimen objects that don't cover a large area, or inthose instances where the stage platform 28 and microscope slide 21 arepositioned to present the specimen 31 in a reasonably flat plane, orwhere a lower power objective is used that has a larger depth of field.For a given level of magnification (such as 10× for example) themicroscope objective 10 with associated video camera is adjusted up ordown, or as in the preferred embodiment, the stage is adjusted up ordown, either adjustment to bring into view an initial reference imageinto the focal plane depth of field of the microscope objective 10 andused to create magnified images of the specimen 31 for a given X, Yposition in the specimen plane. A first series of planar abutted imagetiles are obtained as described in the preferred embodiment as thereference tile set, and stored in the data structure previouslydescribed, and as shown by example in FIG. 3C, wherein the examplereference tiles 43 and 44 are depicted. The reference tiles Z-axisposition in this embodiment are computed using the results of a priorsetup procedure where the Z-axis positions at three separate places on aspecimen are determined and a mathematical Z-axis plane is determinedacross the X, Y plane of the specimen by computations involving fittinga plane in the Z-axis by using three X, Y points with known Z-axisvalues. In this case during the scanning process this computed positionis used instead of the iterative, recursive, automated focus proceduredescribed in the first preferred method. This results in a faster scanand image capture process.

By way of illustration the capture of the complete set of tiles in thisplane may be visualized in cross section as the depth of field 41 inFIG. 4. This scan captures the upper surface of much of the specimen 31.Then, in accordance with this embodiment, the stage Z-axis position ischanged according to one Z Stack increment and another series of imagesare captured and stored in the data structure shown in FIG. 3. Forexample, if the first series of images used the focal distancecorresponding depth of field represented by reference numeral 43, thenby decreasing the stage Z-axis position relative to the microscopeobjective, the next series of images would be represented by referencenumeral 46 a. Conversely, by increasing the stage Z-axis positionrelative to the microscope objective the next series of images would berepresented by reference numeral 45 a. Subsequent series of images canlikewise be captured by positioning different Z-axis planes in the depthof field region of the objective. In the embodiment depicted in FIG. 4,in addition to the original image series captured 43, two other seriesrepresented by reference numerals 46 a and 46 b and two additionalseries represented by reference numerals 45 a and 45 b are alsocaptured. By capturing and storing these additional images fromdifferent regions of the specimen, a virtual focusing capability can berealized as described below in more detail. It may also be realized thatthis method of scanning may be more suited to a type of triple pixelline sensor described above as a 3 by 2098 sensor. Sometimes this isreferred to as a single line sensor. In this case since small discreteindividual tiles are not available, the prediction of a reference planeby computation, or simple assumption of a completely flat and parallelX, Y-plane may be preferred. This type of scan results in saving imagesof longer strip tiles, with a width of 2098 pixels inside the field ofview of the microscope objective 10 in one direction, but the savedimages extending beyond the field of view by continuous scanning andstoring in the other direction. The abutted 2098 pixel wide strip tilestaken together side by side form a virtual microscope slide.

Also as illustrated in FIG. 4, there are two focal depth of fields aboveand two focal depth of fields below, both provided with respect to theinitial reference setting. In a given application, it may be appropriateto provide only a single additional set of images using only oneslightly different focal depth of field (or focal plane) above andbelow. For most purposes, however, images captured at a plurality ofdiffering focal planes are appropriate. In the preferred embodiment,four focal planes above and four focal planes below are used in additionto the original reference focal plane to provide a total of nine sets offocal plane images. Wherein each set of focal plane images correspondsto a given focal distance from the reference setting and all of the setsshare the same level of magnification. By providing this many sets bothabove and below the reference focal plane, relatively smooth anddetailed virtual focusing can be realized that well mimics the look andfeel of focusing with an actual microscope within a useful range offocusing.

As described in the above, the various depths of field substantiallyabut one another. In an alternative embodiment, and as illustrated inFIG. 5, a given depth of field 51 for a given series of images canpartially overlap with another depth of field 52 for a different seriesof images. Or, if desired and as illustrated in FIG. 6, different depthsof field 61 and 62 as corresponding to different image series canneither overlap nor abut one another. Instead, a small gap can existbetween the two fields. In general, adjusting the focal distances suchthat the fields are substantially adjacent one another with little or nooverlap probably represents an optimum configuration, but the otheralternatives may be useful for some purposes depending upon userrequirements.

With reference to FIG. 7, an initial focal plane 71 (as initiallydetermined or predetermined manually or automatically) having acorresponding depth of field 41A may be appropriately used when imaginga particular section of a specimen (not shown) that is within the fieldof view when the microscope is located in a first position 10A. That is,when the microscope is so positioned, this initial focal distance 71represents an optimum focus by whatever standard the user applies. Inaccordance with the various embodiments above, one or more additionalimages are also taken of this same field of view with slightly differentfocal distances. At another portion of the slide, however, when themicroscope is positioned at a second position 10B, it may be that adifferent initial focal plane 72 will yield an optimum focus when usingthe same standard as was applied earlier. This different initial focaldistance 72 will have a corresponding depth of field 41B that issubstantially identical in size to the depth of field 41A for the firstposition's initial focal distance 71 but that is positioned a differentdistance from the slide 21. This is often the case when imaging tissuemicroarray (TMA) cores as described in U.S. Pat. No. 6,466,690 B2(entitled Method and Apparatus for Processing an Image of a TissueSample Microarray). There the image capture is from a great manydifferent objects, TMA cores, arranged over essentially the entiresurface of the glass microscope slide. Therefore, while the resultingimages still comprise a abutted composite representations of the object,they refer to different reference image focal planes. And, according tothis embodiment, regardless of differences as may exist between theinitial focal reference focal plane from object to object, eachresulting image will nevertheless have an identical number of Z Stackfocal planes available for fine focusing by a user.

As discussed above, virtual microscope slides, whether created from manysmall tiles as in the preferred embodiment, or whether created in stripsof line segments, and whether they are stored in a tiled data structureor whether they are stored as one large reconstructed image in memory,such as one focal plane from the set of 5 focal planes 91 in FIG. 9,cannot usually be viewed in their entirety at the original capturedresolution because of the finite size and pixel dimensions of a remoteviewers computer display screen. As depicted in FIG. 8, one prior artapproach that is useful in this regard utilizes a plurality ofindividual images 83, referred to as tiles, to form a larger compositeimage of the slide 81 and the specimen 82. U.S. Pat. No. 6,396,941 B1(entitled Method and Apparatus for Internet, Intranet and Local Viewingof Virtual Microscope Slides), the contents of which are incorporatedherein by this reference, teaches the Internet or intranet display ofvirtual microscope slides. As taught therein, a virtual microscope slidetypically comprises a digitized magnified view of part or all of amicroscope slide and an object (such as a biological specimen) disposedthereon. The aforementioned patent also teaches various Internet serverand thin client, and other Java Applet and ActiveX viewer methodsenabling the reconstruction of the virtual microscope image content fora remote viewer. It will be appreciated that the viewing of a singlefocal plane depth of view is accomplished whether the image is stored asa tiled database structure or as a complete single image plane incomputer core memory. In the preferred embodiment of this inventionhowever, when the remote viewer sends a request to the server for areference image tile focus for a defined region of interest, the serveralso sends the associated Z Stack images all in sequence for that regionof interest. The associated Z Stack images are cached by the localcomputer so that a smooth and rapid local viewing can simulate theanalog optical focusing operation of a real microscope.

Referring now to FIG. 10, in one embodiment, a user can employ astandard computing platform to interface to the virtual slide server anddata storage facility that retains the virtual microscope slideinformation as described above for a given specimen. A standardclient/server model works well to facilitate such a relationship, butother data transfer mechanisms can be used as well as appropriate to agiven application. The relevant process begins with a user platformretrieving 101 a desired image at a particular magnification × (such as,for example, 40×). As described in the aforementioned patent, all imagesfor the object need not be immediately retrieved and made availablelocally. To minimize network transactions, in fact, only the datarequired to display a single field of view need to be immediatelyretrieved and displayed. In the system and method of the currentinvention and the various embodiments above, each field of view has acorresponding plurality of images with each image representing adifferent focal plane. Therefore, when retrieving and displaying thefirst image, one of these images must be selected first. In thepreferred embodiment the selection is the reference image setcorresponding to those tiles that will fill the view window of theremote viewers image display screen. Also, the associated Z Stack imagesfor each reference tile are transmitted and cached in the localcomputer. In one embodiment, where the initial automatically determinedoptimum focal plane image is flanked on each side by four differentfocal plane images, the initial image itself can be automaticallyselected for initial retrieval 101 and display 102. The process thenmonitors 103 for instructions from the user to modify the focus. When nosuch instruction appears, the process continues 104 in accordance withwhatever other functions are supported (for example, input from the userindicating a desire to scroll the image in a particular direction can bereceived and used to cause retrieval and display of correspondingimages). When a focus modification instruction is received, however, theprocess retrieves 105 the image from the local memory cache for thatfield of view that corresponds to the instruction and displays 106 it.Pursuant to one embodiment, the user can be limited to moving the focusin a step by step process with a mouse wheel 93 or keyboard 37 up ordown arrow keys, such that each increment causes retrieval and displayof the next adjacent image in the Z-axis dimension. In the preferredembodiment the user, or remote viewer, can move about and focus on thevirtual microscope slide with a wheeled mouse control, essentially asone moves about and focuses with a physical microscope and slide. Withthis capability, a wide variety of specimens can be readily viewed withgood results. Not only can the resultant virtual microscope slides beused for educational and training purposes, but also for bothqualitative and quantitative analysis purposes in support of variousdiagnostic processes. With reference to FIG. 11, pursuant to oneoptional embodiment, when a user seeks to modify 103 the focus asdescribed, the process can determine 111 whether a focusing limit hasbeen reached. For example, if the user platform has already retrievedand displayed the image that was captured using the focal plane at thefurthest Z-axis dimension from the reference tile and the user is nowinstructing the platform to focus on an even further distance, thepresent display can be maintained 112. Optionally, a text message orother indicator can be provided 113 to the user to alert the user thatthe focus limit has been reached. In another embodiment, a visualindicator can be provided to the user to indicate a present focusingposition within a range of focusing possibilities, such that the usercan ascertain for themselves such a condition.

Those skilled in the art will recognize that a wide variety ofmodifications, alterations, and combinations can be made with respect tothe above described embodiments without departing from the spirit andscope of the invention, and that such modifications, alterations, andcombinations are to be viewed as being within the ambit of the inventiveconcept. It is intended in the appended claims to cover all thosechanges and modifications which fall within the true spirit and scope ofthe present invention.

1. An apparatus for viewing a virtual microscope slide data structurecontaining images of a specimen, the apparatus comprising: a controlsystem configured to retrieve images from the virtual microscope slidedata structure and construct a composite image of a portion of thespecimen based on at least a portion of the first or second set ofimages, the data structure comprising a first and second set ofcontiguous non-overlapping images of the specimen, and coordinatinginformation that relates positions of the first and second set ofimages, wherein a plurality of images in the first set of images areassociated with a first focal plane and a plurality of images in thesecond set of images are associated with a second focal plane, whereinthe second focal plane is displaced relative to the first focal plane.2. The apparatus of claim 1 wherein the control system is configured toretrieve the first and second plurality of images over an Internet orintranet transmission channel.
 3. The apparatus of claim 1 wherein thecontrol system is configured to construct a composite image at the firstfocal plane based on the first plurality of images and the coordinatinginformation.
 4. The apparatus of claim 1 wherein the control system isconfigured to construct a composite image at the second focal planebased on the second plurality of images and the coordinatinginformation.
 5. The apparatus of claim 1 wherein control system isconfigured to receive a focus input and select at least a portion of thefirst or second set for construction of the composite image.
 6. Theapparatus of claim 1 further comprising an Internet browser configuredto manipulate the first and second set of images.
 7. The apparatus ofclaim 6 further comprising an applet configured to manipulate the firstand second set of images.
 8. The apparatus of claim 1 wherein controlsystem is configured generate a first composite image based on the firstset of images and a second composite image based on the second set ofimages, the control system being configured to switch between the firstand second composite images to simulate focusing up or down.
 9. Theapparatus of claim 1 wherein each image in the first set of images isassociated with a unique x, y planar position of the specimen.
 10. Theapparatus of claim 1 wherein the first focal plane and the second focalplane have depths of field that are at least partially overlapping. 11.The apparatus of claim 1 wherein the first focal plane and the secondfocal plane have depths of field that are at least partially abutting.12. The apparatus of claim 1 wherein the first focal plane and thesecond focal plane have depths of field that are non-overlapping.
 13. Anmethod of viewing a virtual microscope slide data structure containingimages of a specimen, the method comprising: retrieving images from thevirtual microscope slide data structure, and constructing a compositeimage of a portion of the specimen based on at least a portion of thefirst or second set of images, the data structure comprising a first andsecond set of contiguous non-overlapping images of the specimen, andcoordinating information that relates positions of the first and secondset of images, wherein a plurality of images in the first set of imagesare associated with a first focal plane and a plurality of images in thesecond set of images are associated with a second focal plane, whereinthe second focal plane is displaced relative to the first focal plane.14. The method of claim 13 further comprising retrieving the first andsecond plurality of images over an Internet or intranet transmissionchannel.
 15. The method of claim 13 further comprising constructing acomposite image at the first focal plane based on the first plurality ofimages and the coordinating information.
 16. The method of claim 13further comprising constructing a composite image at the second focalplane based on the second plurality of images and the coordinatinginformation.
 17. The method of claim 13 further comprising receiving afocus input and selecting at least a portion of the first or second setfor construction of the composite image.
 18. The method of claim 13further comprising: generating a first composite image based on thefirst set of images and a second composite image based on the second setof images; and switching between the first and second composite imagesto simulate focusing up or down.
 19. The method of claim 13 wherein eachimage in the first set of images is associated with a unique x, y planarposition of the specimen.
 20. The method of claim 13 wherein the firstfocal plane and the second focal plane have depths of field that are atleast partially overlapping.
 21. The method of claim 13 wherein thefirst focal plane and the second focal plane have depths of field thatare at least partially abutting.
 22. The method of claim 13 wherein thefirst focal plane and the second focal plane have depths of field thatare non-overlapping.